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Introduction 


The purpose of this paper is to contribute to the corporate memory of 
audit experience. While it records the details relevant to a specific audit project, 


it may be read for the following aspects of general interest: 


- the advantages of a results-oriented approach compared with a 


systems (perhaps better described as procedures-based) ap- 
proach; 


= a simplified approach to the audit of applied research; 
= interviewing by telephone; 
- maintaining a productive relationship with the auditee. 


We therefore describe the specific audit project in enough detail to 
support these general points. The paper concludes with a discussion of how this 
approach will affect the Auditor General's mandate and the productivity of the 


Office's resources. 


The Audit Project 


In 1984 the comprehensive audit of the Department of the Environ- 
ment, which is conducted in phases, concentrated on the Environmental Conserva- 
tion Service. Parks Canada was the focus in 1983. The mandate of the Service is 
to promote the sound management of the three resources: inland waters, wildlife, 
and land. Water management research was chosen as our specific audit project as 
one of the two major activities associated with inland waters. What follows below 
is a description of how the project was designed, how it was executed, what our 
findings and conclusions were and finally, what subsequent analysis we performed 


in order to make relevant recommendations to the Department. 
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Design of the Project 


Through our usual audit planning processes, we reached an under- 
standing that water management research was an activity dedicated to providing 
information needed by managers of water projects serving a national purpose. 
This information could only be obtained by performing applied research. Depart- 
mental management confirmed the view that water management research was 


therefore essentially a service. 


How does the Office audit to determine whether or not a service is 


being provided with due regard for efficiency and economy? 


The usual method we adopt in the Office is to perform a procedures- 
based audit which concentrates on the management processes employed to 
produce the service, such as planning, allocating, acquiring, operating, controlling 
and evaluating. However, we rejected such an approach in this case for several 


reasons. 


First, during the planning phase we observed that the two research 
institutes where the research activity was carried out were particularly skilled at 
managerial processes (smoke and mirrors). Hence we considered that a proce- 
dures-based approach would not allow us to deal readily with what perhaps is one 
of the inherent weaknesses of applied research in Government” that of serving 


the purposes of the researcher more than those of a specific client. 


Also, because of the limited time and resources available, we needed 
to explore alternative audit approaches that promised increased productivity from 


limited OAG resources. 


Finally, since we wanted to report our audit findings and conclusions in 


simple terms to our lay clients we saw a clear need to avoid the technical, 


; ici d Programs for 
. See page 2 of the report "Task Force on Federal Policies an 
By acre Development", July 1984, in which Douglas Wright defines 
"supply-driven" versus "demand-driven" research. 
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complex and esoteric involvement of the researchers. We needed an audit 
approach that did not require either an understanding of the research or of what 


seemed to be very complex managerial processes. 


Through the process of continuous dialogue we obtained the auditee's 
agreement as to the applicability of the simple criterion of use as a test of 
whether or not the research had provided service to the managers of inland water 
projects. We defined use by water managers very broadly as an influence by the 


research on any policies, regulations or operations relating to inland waters. 


The auditee was confident that the research would meet the criterion 
of use. Managers stressed that their research work was of a practical nature 
aimed at solving real problems and not concerned with the purer or basic concern 
of proving why "water is wet". The managers’ acceptance of the use criterion 
implied that they believed value for money (or due regard for economy and 
efficiency) would be demonstrated by a high percentage of use of the research 
products (or outputs) in the management of inland waters. It also suggested that 
they believed the research activities were intimately involved with water . 


management problems. 


The design of a results-oriented audit rapidly evolved, with the 
objective of determining whether or not the products or outputs of the research 


activity were used by the managers of inland waters. 
Our detailed audit program had the following major components: 


= a questionnaire to solicit the researcher's view of use (copies are 


available from the authors); 


. a questionnaire to solicit the user's view of the research (copies 


are available from the authors); and 


- a means of sampling from among the research project population 


for the purpose of completing the two questionnaires. 
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The questionnaires were designed to make the researcher the focus of 
the review. They also allowed OAG staff to remain neutral in that they did not 
have to pass judgement as to the nature of the research or the nature of its use. 
They allowed us to determine the population of potential users; in fact, the 
researcher could lead us to all users (potential and actual) and give us a total 
listing of use and impact. They also allowed the user (or potential user) to say 
whether the research product had been or was likely to be used and explain the 


nature and impact of use. 


In choosing a representative sample of the research project population 
to complete the questionnaires, the following considerations were taken into 


account: 


the diverse nature of the population -- each project was dif- 


ferent in size, type and the issues; 

= the finite time available for determining use; and 

= the possibility that the researcher's and user's memories of 
projects completed and used in the past might be somewhat 


dimmed. 


t was therefore decided initially to subject all the research projects 


_— 


that were under way in fiscal 1982-83 to the questionnaires. The total population 
for 1982-83 appeared to be suitable for the resources which we had available to 
carry out the review. Also, having all the projects within one fiscal year would 
reduce concerns that could be raised by sampling from research populations in 
several fiscal years. The choice of fiscal 1982-83 was a compromise between 


allowing too little time for use and allowing too much time, so that memories of 


the researcher and user could be dimmed. 
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Execution of the Project 


Having identified the research projects (and the key researchers) for 
1982-83, we began with the questionnaire designed to gain the researcher's view 
of research use. These were initially completed through personal interviews, 
involving only one member of OAG staff and taking one to one and a half hours. 


The major outputs from each completed questionnaire were as follows: 


= general understanding of the research product, elicited through 
carefully structured questions and the various classifications in 


the questionnaire; 


= a list, with telephone numbers whenever possible, of actual and 


potential users; and 


- a completed questionnaire signed off for substantive content by 


the interviewer and interviewee. 


Telephone interviews were conducted with equal success. A copy was 
mailed out for sign-off. Interview times were approximately the same but we 
saved considerably on set up times, since most of the personal interviews involved 
travel to locations throughout the national capital region or to Burlington, 


Ontario. 


Using the referrals provided by the researchers, we completed the 
questionnaires to solicit the user's view of the research, mainly through telephone 
interviews. Because the completed "researcher's view of use" questionnaire gave 
us referrals to several users, telephone interviews were essential. The users 
(actual and potential) were distributed throughout the North American continent 
and a few eee in Europe. Copies of these completed questionnaires were also 


mailed out for sign-off. The user's (potential and actual) view of research use, 


provided by the completed questionnaire, yielded the following outputs: 


a corroboration of our lay understanding of the research product 


established through the " researcher's view of use" questionnaire; 
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= a description of the nature of the use (actual or potential); and 


= a completed questionnaire signed off for substantive content by 
the interviewer and interviewee. 


Audit Findings and Conclusions 


Although we intended to review the total population of the research 
projects that were under way in fiscal 1982-83, the review was halted after 
Sampling approximately one quarter of the population (i.e., 52 projects). We 
considered that this sample was a reasonable and fair representation of the 
population, due to the confirmation, by the later part of the sample, of a pattern 


that was rapidly established during the early part of the sample. 


Any concern about a bias in our sample against the research products 
could be balanced by the factors that represented a bias somewhat in favour of 


the research projects. 


We clearly signaled our intention to solicit use of the research projects 
or products to the two research institutes during our set-up work (designing and 
testing of the questionnaires). All the research staff had been circulated by the 
management of the institutes to prepare for our detailed review. They were 
therefore well prepared for us and aware of our intention to review the use of 
their research work. Further, our detailed questionnaire could be considered to be 


an instrument for eliciting as opposed to suppressing considerations of use. 


Also, our interpretation of use could be considered generous in that we 
did not impose any qualifications as to the quantity or quality of use. Since the 
questionnaire review process generated up to four potential users or uses, any 
confirmation by any one of these referrals that the research had affected or 
influenced any policy, regulation, or operation was accepted as use. We also 


accepted that the effect or influence could be produced by any research product 
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such as a report, a publication, a mathematical model, a method, an instrument or 
a consultation (verbal or written). 


Before discussing our audit findings and conclusions we must stress 
that our use criterion was rudimentary in that it was not eoncerned with the 
quality of use but only with the fact of use. This was simply that it had been used 
in some way and somewhere by managers of water projects. Our testing of water 
management research in this manner is akin to testing a repair shop for diesel 
engines for ice breakers. The test was not concerned with the quality of the 
service that had been performed on the engines. I merely tested that the work 
that had been done in the repair shop was indeed on ice breaker diesel engines. 
Such testing is remote from program effectiveness evaluation; we were not 
concerned with how good the diesel driven ice breakers were at breaking ice, 


creating jobs or fostering international relations. 


Two types of findings were obtained: quantitative findings relating to 
use; and qualitative findings relating to the characteristics of the research 
projects. On the basis of the quantitative findings relating to use, we rapidly 
concluded that the water management research activity did not meet the use 
criterion. The qualitative findings relating to the characteristics of the research 


projects gave us a background for subsequent analysis, which is described below. 


Analysis of the Findings 


Looking back, we can see that the powerful advantages of the results- 
oriented approach emerged at this point. It had enabled us to get rapidly at the 
major issue in the water management research activity, i.e., the research was not 
connected to the needs of the intended client. It had also given us some 
important ateniGe information that suggested where the weaknesses were in 


the management of research. Theoretically we now had two alternatives. 


We could devise a very selective procedures-based audit to examine 


the aspects of management most likely to be related to the major issue we had 
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identified. Or we could pin-point management weaknesses by a "desk top" 
comparison with other applied research management systems considered by the 
literature or experts to be good and appropriate. 


We selected the latter alternative in view of the limited time and 
resources available. For the sake of completeness, we are including the essence 


of our "desk top" analysis. 


Our findings. Our qualitative findings provided the following "typical" 


or "most likely" major characteristics of the research projects: 


= the research project was in the area of the researcher's post- 


graduate specialization; 


- the research related specifically to some aspect of the Great 


Lakes (close to where the major research institute was located); 
= it had received budget support for five years; 


> the results of the research tended to be more problem revealing 


rather than problem solving; 
= the research had not yet influenced water management; 


= the research had yielded many papers published in reputable 


professional journals; 
= the research work was original; and 


= it had enabled the researcher to be recognized by his peers as an 
authority in a specialized area. 


Basic research. We found in our survey of the current literature on 


research management and through consultation with experts and authorities, that 
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the following were likely to be the important characteristics of basic research. 


Such research is: 


teristies of 


- predominantly concerned with advancing the frontiers of scienti- 


fie knowledge for understanding; 
= not concerned with the utility of the results; 


- not concerned with social, political, economic implications or 


effects of the research; 
- initiated from the bottom up; 
= done by a researcher working alone rather than in a team; 


- judged by peers who learn of it through publication in learned 


journals; 
A available to the world community; and 


a conducted on a long-time scale with little pressure to complete 


or finish. 


Applied research. In contrast, the following emerged as the charac- 


applied research: 


- predominantly concerned with the utility of new knowledge; 


intimately associated with trying to understand actual needs and 


practical problems, making it more difficult to design clear-cut 


experiments; 


* usually a team activity; 
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- concerned with social, economic, political effects or implica- 
tions; 


- judged more by the utility of the research product than by peer 
acceptance; 


= not usually widely available through publication; 


= initiated from the top down, with priorities and pressure to 


produce; and 
= conducted over much shorter time than basie research. 


Comparing our findings with the two categories of characteristics, it 
was easy to diagnose the reason for the low use of the research. The research 
activity that was being conducted at the two inland water research institutes was 
more basic than applied contrary to what our audit planning work had led us to 


believe it should be. 


Departmental management was very receptive to our recommenda- 
tions that the management in the two research institutes should be modified to 
favour applied rather than basic research. The ADM responsible for water 
management research confirmed in writing his agreement with our methodology, 


findings and recommendations. 


The Task Force on Federal Policies and Programs for Technical 


Development makes the following relevant observations: 


The most effective research and development, we believe, is 


'demand driven', where the research Is undertaken in response to a 


clearly defined need.... The least effective research and development 


is 'supply-driven', where the research institutions, rather than an 


external market, define the problem and, at their own speed, seek 


oka goh foal dantearey 904 


wienisoe Mn . ‘ Bisex 


) @mnee See SAIN ie 


: 


Siew ie@er ie Ah Sie ae senagget ingen 


wip ° 7 pera re Wey f we 
Ge Oodlion wy Mle hate 


maine) hs) Oi) vyenS bre, sawiigy 


-ll- 


solutions. Sometimes they come up with brilliant solutions for which 


there is no problem... products for which there is no market. 


.-. We believe that closer relationships with end users of the research 


is the best prescription for their continued vitality... . 


Advantages of the Results-oriented Approach 
Over the Procedures-based Approach 


With some temerity, we have characterized the results-oriented 


approach as having four distinct stages. 


The first stage is a review to test, on the basis of simple criteria, the 
inherent risk of the activity or the risk associated with the activity. This is a test 
of the major output of the activity or of the vital connection that it has or should 
have to other activities. In the project we used for an example, the inherent risk 
of applied research is that it may tend to become an end in itself, rather than a é 
means to an end. Therefore, the criteria of use was a test of whether or not the 
applied research activity was connected to the intended clients. The following 


are other examples of areas that lend themselves to this approach: 


Staff Functions. The inherent risk is that staff functions are not 
connected to or used by operations or management. The simple 


criterion of use would be applicable here. 


Libraries and Technical Information Activities. The inherent risk 


again is lack of use by intended clients. 


Program Evaluation. Notwithstanding the existing formal procedures- 


based approach of the OAG, the inherent risk is the lack of use by 


senior management. 


The second stage is to assess the quantitative findings of the first 


stage in the context of the qualitative findings which will have been observed as a 


-12- 


by-product of the first stage. Should the activity not meet the established 
criteria, the qualitative findings will assist in the design of a selective proce- 
dures-based audit to deal with those aspects of the management system that could 


be weak and hence responsible for the criteria not being met. 


The third stage is to execute the selective procedures-based audit. As 
discussed earlier, this can be either a "desk-top" review or a detailed field review 
of the particular management system or Systems that are suspected of being 
weak, 


The fourth stage is reporting. In our experience and judgement, it is 
not necessary to report the conclusions in the same order as the audit was carried 
out. The results-oriented audit approach could have revealed that the activity 
was not producing an appropriate output or was not appropriately connected; 


however, it could be reported as a specific management weakness. 


What occurs to us, as a further general observation, is that the Office 
is not bound by its legislative mandate to a particular way of carrying out its 


audit work; our legislative mandate only spells out how and what we report. 


Although we fully appreciate that a results-oriented approach which 
progresses from an activity to a program and finally to an entity could be 
tantamount to an assessment of effectiveness, this is permissible providing that 
we do not report directly on effectiveness. If we use the results-oriented 
approach in our audit work as a means of quickly identifying and diagnosing 
management system weaknesses, and subsequently only report on the system 


weakness, we have stayed entirely within the mandate of the Office. 


While there are government programs that are not suited to a results- 


oriented approach and where the measurement of effectiveness is extremely 


difficult, there are many programs where results could be measured relatively 


simply. The following are examples of such areas. 


Weather Forecasting. The inherent risk is that the activity is pursuing 


the advancement of meteorological science rather than producing 
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outputs relevant to its stated priorities of safety of life, security of 
property, and increased efficiency of weather-dependent economic 
activities. 


Home Insulation Program. The inherent risk is that the program does 


not achieve significant benefits compared to cost. 


Assisted Housing. The inherent risk is that the intended clients were 
not served and that those who were served experienced excessively 


rich housing benefits. 


Canada Post Corporation. The inherent risk is that mail service is now 
inferior to other means of communication. The monopoly of Canada 


Post is no longer relevant. 


Unemployment Insurance. The inherent risk is that such payments 


encourage the status of "Unemployed". 


Agriculture Canada. The inherent risk is that the programs in 


agriculture are not tailored to the current needs of Canadian agricul- 


ture. 


Veterans Affairs. The inherent risk is that the services currently 


provided are an inferior duplication of social services currently 


available from a variety of sources. 


In our opinion, the following are the significant advantages of the 


results-oriented approach over comprehensive auditing (while accepting the fact 


that there are activity areas where it cannot be used). 


® [It enhances the productivity of OAG resources: 
[t allows rapid identification of the major issues. 


It allows OAG resources to be focused selectively on 


management system weaknesses. 
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& It increases the professional standing of the OAG staff because: 


- It is more acceptable to the auditee than a procedures- 


based audit in that its purpose and language are simple. 


- It is more likely to leave behind a useful management tool 


and perspective to the auditee. 
= It allows the preparation of simple management reports. 


= The evidence is usually compelling in that it is dealing 
directly with the results of the activity. A systems- 
based approach is at best a proxy measure of due 
regard -- the existence of the best procedure does not 


guarantee due regard. 


e It may allow a more acceptable form of reporting to Parliament 
program effects (or indeed program evaluation) where questions 
of policy are clearly not being questioned. For example, in the 
ease of Canadair, the Office does not question the desirability of 
keeping Canadian Aerospace workers employed, but it can report » 


that the output of Canadair is not a marketable product. 


A Simplified Approach to Auditing Applied Research 


There is no doubt, in our opinion and experience, that applied research 
is most amenable to a results-oriented audit approach. We believe it is both far 
more productive and easier to execute than a procedures-based approach. Since 


there are many resources in the federal government devoted to applied research, 


we recommend that our approach be adopted, with suitable variations to tailor 


questionnaires to the specific characteristics of the applied research activity. 


The four sections of these questionnaires are: 


1 A descriptive section used to obtain tombstone data such as 


project name, how initiated, when initiated, when completed, 


together with a general description of the actual research. 
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De A substantive section to elicit a description of the research 
product; the major conclusions; the major recommendations; the 
implications of these; the classification of the area of research 
affected; an assessment of the value of this research to the 
researcher, the user and to Canada; a description of the user and 
the use to which it is being put; and a classification of the use, 


as opposed to the user. 
3. A sign-off section. 


4, A budgetary section. 


Interviewing by Telephone 


One of the by-products of our audit project was the discovery that our 
productivity could be increased (with no significant detriment to the integrity of 
the audit project) by using the telephone to conduct interviews. Although our 
experience was all derived from one audit project, we believe that the following 


generalizations are possible: 


= With a well structured questionnaire, the same type of informa- 


tion can be collected by telephone as by person-to-person 


interviews. 


Considerable savings in travel time and set-up time are possible. 


Having to obtain sign-off by mail, although admittedly dependent 


on the Postal Service, was not considered to be a disadvantage. 


Normally, person-to-person interviews would require the use of 


the mail to exchange formal minutes or records of the substan- 


tive aspects of the meeting. [n our project, we obtained the 


required sign-off in every telephone interview. 
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- There was a consensus among the three auditors who conducted 
telephone interviews that the telephone afforded a sense of 
privacy to both parties. Exchanges were facilitated and con- 
fidences more readily volunteered than in person-to-person 
situations. Subsequently, we learned that this had been esta- 
blished in social science research, where the telephone was used 


to interview subjects in delicate or personal areas. 


< The telephone also afforded an opportunity for carrying out rapid 
follow-up when matters could not be dealt with during the course 


of a single interview. 


Maintaining a Productive Relationship with the Auditee 


From a lessons-learned perspective, we consider that the most univer- 
sally transferable aspect of this project was our ability to maintain clear 
communication channels between auditor and auditee from audit project design , 
through to final reporting. The following outlines some of the steps in establish- 


ing this communication. 


Nig The auditee was apprised of our audit interest before we began 


the audit project design. 


2. We held formal sessions with the auditee to exchange options for 


criteria and audit approach. 


5 We obtained consensus with the auditee both on the audit 


criterion and the audit approach. 


4 The auditee was involved with the design of questionnaires, the 


testing and a pilot project to determine exactly how the audit 


project would be executed. 
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5. We discussed the early results and findings of the audit with the 
auditee. 


6. Before preparing the project report, we made a comprehensive 
presentation to the auditee covering our findings, conclusions 


and likely recommendations. 


he At each of the above stages, we modified our audit work to 


reflect the results of dialogue with the auditee. 


The greatest benefits of maintaining a productive relationship with the 
client were those we received during the course of our final debriefing with senior 
management. Although our findings and conclusions were not positive for the 
ADM responsible, he defended our methodology, findings and recommendations 


when they were criticized by other ADMs. 


Impact of the Approach on the OAG Mandate 
and the Productivity of the Office 


Although we invite discussions on the impact of the results-oriented 


approach, our views are, succinctly, as follows: 


8 Our mandate relates to what we report to Parliament, not to 
how we conduct our audit work. We maintain that, although the 
results-oriented approach may in certain cases amount to an 
implicit evaluation of the effectiveness of an activity or pro- 
gram, we would report our findings as comment on management 
systems. Our evaluation would be the evidence for our opinion 


on due regard for economy and efficiency by management. 


e Since the results-oriented approach allows for a rapid identifica- 


issues and subsequent design of a selective 


tion of major ) 
we contend that it offers 


procedures-based audit approach, 


substantial potential for increased office productivity. 
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Conclusion 


The results-oriented approach to comprehensive audit has, in certain 


areas, significant potential benefits for the Office of the Auditor General. There 


is therefore a need to experiment with this approach on a broader scale and to 
collect a greater depth of experience. 


